Skip to content

feat: metrics reporting for scan and commit#589

Open
evindj wants to merge 10 commits intoapache:mainfrom
evindj:metrics_core_components
Open

feat: metrics reporting for scan and commit#589
evindj wants to merge 10 commits intoapache:mainfrom
evindj:metrics_core_components

Conversation

@evindj
Copy link
Copy Markdown
Contributor

@evindj evindj commented Mar 11, 2026

Initial commit for addressing #533

@evindj evindj marked this pull request as draft March 11, 2026 20:33
@evindj evindj force-pushed the metrics_core_components branch from 5fa02cb to 0a85180 Compare March 11, 2026 20:48
@wgtmac
Copy link
Copy Markdown
Member

wgtmac commented Mar 20, 2026

Let me know when ready for review :)

@evindj
Copy link
Copy Markdown
Contributor Author

evindj commented Mar 21, 2026

Let me know when ready for review :)

for sure ! most likely tomorrow

@evindj evindj force-pushed the metrics_core_components branch from eaacbe8 to 5241044 Compare March 25, 2026 02:56
@evindj evindj force-pushed the metrics_core_components branch from 5241044 to d26fb96 Compare March 25, 2026 03:15
@evindj evindj marked this pull request as ready for review March 25, 2026 04:08
@wgtmac
Copy link
Copy Markdown
Member

wgtmac commented Mar 27, 2026

Thanks for updating this! I still need to take some time to get familiar with the Java implementation and its API in the rest spec before reviewing it.

Copy link
Copy Markdown
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! I have some general questions on the design, especially on the capability of customization. IMO we can focus on the API design for now and later integrate with other classes. Please let me know what you think.

Comment thread src/iceberg/metrics_reporter.h Outdated
struct ICEBERG_EXPORT ScanReport {
/// \brief The fully qualified name of the table that was scanned.
std::string table_name;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why some metrics have blank lines in between but others don't? I think we can remove all these blank lines to be compact.

Comment thread src/iceberg/metrics_reporter.h Outdated
int64_t snapshot_id = -1;

/// \brief Filter expression used in the scan, if any.
std::string filter;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be std::shared_ptr<Expression>?

Comment thread src/iceberg/metrics_reporter.h Outdated
namespace iceberg {

/// \brief Duration type for metrics reporting in milliseconds.
using DurationMs = std::chrono::milliseconds;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hard-codes our reported duration unit to be milliseconds, which violates the spec I think?

Comment thread src/iceberg/metrics_reporter.h Outdated
std::string filter;

/// \brief Schema ID.
int32_t schema_id = -1;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are missing some fields like projectedFieldIds and projectedFieldNames from the Java implementation. I think they are required by the REST spec: https://github.com/apache/iceberg/blob/149cc464f9b7df800cc5718af725983473819504/open-api/rest-catalog-open-api.yaml#L3990-L4023

Comment thread src/iceberg/metrics_reporter.h Outdated
std::string table_name;

/// \brief The snapshot ID created by this commit.
int64_t snapshot_id = -1;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use kInvalidSnapshotId defined in the iceberg/constant.h

Comment thread src/iceberg/metrics_reporter.h Outdated
/// \brief Total number of data manifests.
int64_t total_data_manifests = 0;

/// \brief Number of data manifests that were skipped.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is inconsistent.

Comment thread src/iceberg/metrics_reporter.h Outdated
int64_t skipped_data_files = 0;

/// \brief Number of data manifests that were skipped.
int64_t skipped_delete_files = 0;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Comment thread src/iceberg/metrics_reporter.h Outdated
int32_t schema_id = -1;

/// \brief Total duration of the entire scan operation.
DurationMs total_duration{0};
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove this as this is not defined by the Java implementation?

Comment thread src/iceberg/metrics_reporter.h Outdated
///
/// This variant type allows handling both report types uniformly through
/// the MetricsReporter interface.
using MetricsReport = std::variant<ScanReport, CommitReport>;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we define MetricsReport as a std::variant, we cannot support customizing metrics report. For example, engines may have more metrics to report than defined by the Java implementation. Even the REST spec does not explicitly define what keys are required.

Instead, should we define the MetricsReport like below?

struct MetricsReport {
  std::string kind;  // can be "scan" or "commit", or whatever customized
  std::unordered_map<std::string, CounterResult> counter_results;
  std::unordered_map<std::string, TimerResult> timer_results;
};

What do you think?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After thinking more on this, I'm fine to use your current approach to define ScanReport and CommitReport with explicit fields. MetricsReports are collected by this library so users do not have the flexibility to customize them. We only need to customize MetricsReporter.

Copy link
Copy Markdown
Contributor Author

@evindj evindj Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the bag of keys approach because it provides for customization without necessarily having to require compilation. let me think a bit more about the entire thing, I might reach out on slack for a quick sync.

///
/// \param reporter_type Case-insensitive type identifier (e.g., "noop").
/// \param factory Factory function that produces the reporter.
static void Register(std::string_view reporter_type, MetricsReporterFactory factory);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we support the Java parity CompositeMetricsReporter? It would be useful if we want to report metrics to multiple sinks.

@evindj
Copy link
Copy Markdown
Contributor Author

evindj commented Mar 31, 2026

Thanks for working on this! I have some general questions on the design, especially on the capability of customization. IMO we can focus on the API design for now and later integrate with other classes. Please let me know what you think.

This is fair, I just wanted to show the end to end picture for ease of understanding.

@evindj evindj marked this pull request as draft March 31, 2026 13:59
@evindj evindj marked this pull request as ready for review April 2, 2026 05:08
# specific language governing permissions and limitations
# under the License.

iceberg_install_all_headers(iceberg/metrics)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing meson.build equivalent for this subdirectory

Copy link
Copy Markdown
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up findings from a deeper design/parity pass over the metrics reporting changes.

committed_ = true;
ctx_->table = std::move(commit_result.value());

ReportCommitMetrics();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Transaction::Commit() always calls ReportCommitMetrics() after a successful metadata commit, even for metadata-only transactions. That helper then reads current_snapshot() from the post-commit table and reports it as if this transaction created that snapshot, which can re-emit stale snapshot metrics and a misleading operation name. Java only reports commit metrics from snapshot-producing commits (CreateSnapshotEvent in SnapshotProducer), so the reporting hook should move closer to snapshot-producing updates or otherwise prove that this transaction actually created a new snapshot before emitting a CommitReport.


// Load metrics reporter from catalog properties
std::shared_ptr<MetricsReporter> reporter;
auto reporter_result = MetricsReporters::Load(final_config.configs());
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The REST catalog now loads a local reporter from catalog properties, but it never constructs the built-in REST reporter for the /metrics endpoint and never combines the two. As a result, scan/commit reports from tables loaded through RestCatalog are never POSTed back to the server even when the server advertises ReportMetrics support. Java's RESTSessionCatalog explicitly combines the catalog reporter with RESTMetricsReporter, so this leaves the feature only partially implemented on the C++ side.

/// \brief Infer the reporter type from properties.
std::string InferReporterType(
const std::unordered_map<std::string, std::string>& properties) {
auto it = properties.find(std::string(kMetricsReporterImpl));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loader treats metrics-reporter-impl as a lowercase registry key and defaults to noop, while Java and the Iceberg docs treat the same property as a fully qualified class name whose instance is initialized with catalog properties. On top of that, the catalog constructors ignore Load() errors and Table falls back to noop, so a Java-style config string or typo quietly disables metrics instead of surfacing a configuration error. That is a public-contract drift, not just an implementation detail.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while Java and the Iceberg docs treat the same property as a fully qualified class name whose instance is initialized with catalog properties

the divergence is because c++ does not support reflection. I decided on making things local case to reduce the risk of typos based on caps on/off.

typo quietly disables metrics instead of surfacing a configuration error
I will update to surface a configuration error.

///
/// Embedded in ScanReport and populated by DataTableScan after PlanFiles()
/// completes. Mirrors the fields in Java's ScanMetricsResult.
struct ICEBERG_EXPORT ScanMetrics {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ScanReport embeds a plain ScanMetrics struct of raw integers and std::chrono::nanoseconds, but Java's contract is layered: live counters/timers are collected in ScanMetrics, converted into ScanMetricsResult, and then wrapped by ScanReport. That layering preserves units, optional presence, noop semantics, and serialization behavior. In C++, every metric defaults to 0, so consumers cannot distinguish "not collected" from a real zero value, and the report shape is no longer compatible with Java's richer contract.

metric.total_equality_deletes = parse_int64(SnapshotSummaryFields::kTotalEqDeletes);
metric.added_dvs = parse_int64(SnapshotSummaryFields::kAddedDVs);
metric.removed_dvs = parse_int64(SnapshotSummaryFields::kRemovedDVs);
metric.created_manifest_count = parse_int64(SnapshotSummaryFields::kManifestsCreated);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CommitMetrics exposes replaced_manifest_count, but ReportCommitMetrics() never reads SnapshotSummaryFields::kManifestsReplaced into it. Java's CommitMetricsResult.from(...) does populate the corresponding manifestsReplaced field from the snapshot summary. Any consumer relying on this report will currently observe 0 even when the commit actually replaced manifests.

Comment thread src/iceberg/table_scan.cc
for (const auto& task : tasks) {
report.scan_metrics.total_file_size_in_bytes +=
task->data_file()->file_size_in_bytes;
for (const auto& del_file : task->delete_files()) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scan report counts equality and positional delete files, but it never increments indexed_delete_files and it never separates deletion vectors from ordinary positional deletes. Java's ScanMetricsUtil.indexedDeleteFile(...) increments all three counters (indexedDeleteFiles, positionalDeleteFiles/dvs, equalityDeleteFiles). Because the C++ report schema already exposes indexed_delete_files and dvs, leaving them unset produces parity drift and misleading metrics for DV-heavy tables.


namespace iceberg {

/// \brief Metrics collected during a table commit (snapshot creation).
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CommitReport only carries summary-derived counters, but Java's commit reporting model also has live commit metrics (totalDuration, attempts) collected through CommitMetrics and serialized through CommitMetricsResult. Because those fields are absent from the C++ public model entirely, even a future implementation cannot fully mirror Java's commit report without another API change.

Comment thread src/iceberg/table_scan.h
/// \param io FileIO instance for reading manifests files.
/// \param reporter Optional metrics reporter for scan metrics.
/// \param table_name Optional table name for metrics reporting.
static Result<std::unique_ptr<TableScanBuilder<ScanType>>> Make(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The builder stores a reporter internally, but there is no public scan-level API to add an extra reporter the way Java's Scan.metricsReporter(...) does. That means C++ can only use the reporter injected from the table/catalog path, while Java supports composing an additional reporter for a particular scan. For observability integrations and tests, that is a meaningful flexibility gap.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch will address.

@wgtmac
Copy link
Copy Markdown
Member

wgtmac commented Apr 17, 2026

I have a design question about the ScanReport / CommitReport model in this PR. What is the intended direction here with respect to the Java metrics API? In Java, the reporting model is layered around metrics collection/result types, which also carry unit information and are backed by reusable measurement utilities (for example counters/timers and the related conversion helpers). In this C++ version, the reports currently look more like raw result structs, and I do not yet see an equivalent measurement abstraction/tooling layer behind them. Is that divergence intentional for C++, or is this PR meant as an initial step toward a closer alignment with the Java design over time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants